Image sensor with image receiver and automatic image switching

ABSTRACT

Provided are an image sensor with one or more image receivers for image switching, and an imaging system and method therefor. The image sensor includes an image sensor array to generate first image data for a first image; a receiver to receive, into the image sensor, second image data for a second image; an image selection circuit coupled to the image sensor array and the receiver to receive the first image data and the second image data and select one of the first image data and the second image data according to one or more image selection criteria and at least one of the first image data and the second image data; and a transmitter coupled to the image selection circuit to transmit the selected one of the first image data and the second image data from the image sensor.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 17/130,371, filed Dec. 22, 2020, entitled “IMAGE SENSOR WITHIMAGE RECEIVER AND AUTOMATIC IMAGE SWITCHING,” which is a continuationof U.S. patent application Ser. No. 16/251,239, filed Jan. 18, 2019, nowU.S. Pat. No. 10,892,287, entitled “IMAGE SENSOR WITH IMAGE RECEIVER ANDAUTOMATIC IMAGE SWITCHING,” the disclosures thereof incorporated byreference herein in their entirety.

TECHNICAL FIELD

The present disclosure generally relates to the field of solid-stateimage sensor arrays, particularly to solid-state image sensor arrayshaving receivers for receiving additional images.

BACKGROUND

Digital cameras, scanners, and other imaging devices often use imagesensors, such as charge-coupled device (CCD) image sensors orcomplementary metal-oxide-semiconductor (CMOS) image sensors, to convertoptical signals to electrical signals for storage, processing, display,and the like. Recently, imaging devices have begun to include multipleimage sensors, for example to capture images using multiple spectra,multiple subjects, and the like.

SUMMARY

In general, one aspect disclosed features an imaging system comprising:an image sensor comprising a first image sensor array to generate firstimage data for a first image, a receiver to receive, into the imagesensor, second image data for a second image, an image selection circuitcoupled to the first image sensor array and the receiver to receive thefirst image data and the second image data and select one of the firstimage data and the second image data according to one or more imageselection criteria, and at least one of the first image data and thesecond image data, and a transmitter coupled to the image selectioncircuit to transmit the selected one of the first image data and thesecond image data from the image sensor; and a second image sensor arraycoupled to the image sensor to generate the second image data for thesecond image.

Embodiments of the imaging system may include one or more of thefollowing features. Some embodiments comprise an image signal processorto process the selected one of the first image data and the second imagedata into a processed image. Some embodiments comprise a display todisplay the processed image. In some embodiments, the first image andthe second image are captured substantially concurrently. In someembodiments, the first and second image sensor arrays possess differentimage capture characteristics, wherein the image capture characteristicsinclude at least one of: pixel size; sensitivity; read noise; gain;exposure time; and spectrum. In some embodiments, the one or more imageselection criteria include at least one of: noise; dynamic range;resolution; brightness; a quantity of features; a quality of features; apresence of text; facial recognition; and facial expression recognition.In some embodiments, the receiver is a Mobile Industry ProcessorInterface (MIPI) receiver; and the transmitter is a MIPI transmitter.

In general, one aspect disclosed features an image sensor comprising: animage sensor array to generate first image data for a first image; areceiver to receive, into the image sensor, second image data for asecond image; an image selection circuit coupled to the image sensorarray and the receiver to receive the first image data and the secondimage data and select one of the first image data and the second imagedata according to one or more image selection criteria and at least oneof the first image data and the second image data; and a transmittercoupled to the image selection circuit to transmit the selected one ofthe first image data and the second image data from the image sensor.

Embodiments of the image sensor may include one or more of the followingfeatures. In some embodiments, the first image and the second image arecaptured substantially concurrently. In some embodiments, the firstimage includes a scene; and the second image includes the scene. In someembodiments, the image sensor array generating the first image data forthe first image, and an image sensor capturing the second image, possessdifferent image capture characteristics, wherein the image capturecharacteristics include at least one of: pixel size; sensitivity; readnoise; gain; exposure time; and spectrum. In some embodiments, the oneor more image selection criteria include at least one of: noise; dynamicrange; resolution; brightness; a quantity of features; a quality offeatures; a presence of text; facial recognition; and facial expressionrecognition. In some embodiments, the receiver is a Mobile IndustryProcessor Interface (MIPI) receiver; and the transmitter is a MIPItransmitter. Some embodiments comprise a single integrated circuitcomprising the image sensor.

In general, one aspect disclosed features a method for an image sensor,the method comprising: generating, in an image sensor array of the imagesensor, first image data for a first image, responsive to receivinglight; receiving into the image sensor, second image data for a secondimage; selecting one of the first image data and the second image dataaccording to one or more image selection criteria and at least one ofthe first image data and the second image data; and transmitting theselected one of the first image data and the second image data from theimage sensor.

Embodiments of the method may include one or more of the followingfeatures. In some embodiments, the first image and the second image arecaptured substantially concurrently. In some embodiments, the firstimage includes a scene; and the second image includes the scene. In someembodiments, the image sensor array generating the first image data forthe first image, and an image sensor array capturing the second image,possess different characteristics, wherein the characteristics includeat least one of: pixel size; sensitivity; read noise; gain; exposuretime; and spectrum. In some embodiments, the one or more image selectioncriteria include at least one of: noise; dynamic range; resolution;brightness; a quantity of features; a quality of features; a presence oftext; facial recognition; and facial expression recognition. In someembodiments, the second image is received according to a Mobile IndustryProcessor Interface (MIPI) specification; and the selected image istransmitted according to the MIPI specification.

BRIEF DRAWINGS DESCRIPTION

The present disclosure describes various embodiments that may beunderstood and fully appreciated in conjunction with the followingdrawings:

FIG. 1 illustrates a conventional imaging system using multiple imagesensor arrays.

FIG. 2 illustrates an imaging system that includes an image sensorhaving an image data receiver and automatic image switching according tothe technology disclosed herein.

FIG. 3 illustrates two image sensor arrays having the same opticalformat but different image capture characteristics.

FIG. 4 illustrates a process that may be performed by the imaging systemof FIG. 2 according to embodiments of the disclosed technology.

DETAILED DESCRIPTION

The present disclosure describes embodiments with reference to thedrawing figures listed above. Persons of ordinary skill in the art willappreciate that the description and figures illustrate rather than limitthe disclosure and that, in general, the figures are not drawn to scalefor clarity of presentation. Such skilled persons will also realize thatmany more embodiments are possible by applying the inventive principlescontained herein and that such embodiments fall within the scope of thedisclosure which is not to be limited except by the claims.

FIG. 1 illustrates a conventional imaging system using multiple imagesensor arrays. Referring to FIG. 1, the imaging system 100 includes aplurality of image sensor arrays 102 a through 102 n, and an imageprocessor 104. Each image sensor array 102 generates image dataresponsive to receiving light 110, for example such as light reflectedby, or radiated by, a subject. The image processor 104 includes an imagesignal processor 106 for processing the image data generated by theimage sensor arrays 102. In the conventional imaging system 100 of FIG.1, the image processor 104 must include multiple receivers (RX) 108 athrough 108 n to receive the image data from the multiple image sensorarrays 102 a through 102 n. In particular, the image processor 104 mustinclude a separate receiver 108 for each image sensor array 102.

FIG. 2 illustrates an imaging system that includes an image sensorhaving an image data receiver and automatic image switching according tothe technology disclosed herein. Referring to FIG. 2, the imaging system200 includes an image sensor 214, an image processor 204, a display 216,and one or more external image sensor arrays 212 a through 212 n. Theexternal image sensor arrays 212 a through 212 n are referred to hereinas “external” because they are external to the image sensor 214. Theimage processor 204 includes an image signal processor 206, and a singlereceiver 228.The image sensor 214 includes an internal image sensorarray 202, one or more receivers 208 a through 208 n, a singletransmitter (TX) 218, an image selection circuit 222, and imageselection criteria 220. The image sensor 214 may be fabricated as asingle integrated circuit. The image sensor arrays 202, 212 may befabricated using any suitable technology. For example, the image sensorarrays 202, 212 may be fabricated as arrays of CMOS CCDs. The imagesensor arrays 202, 212 may all be fabricated using the same technology,or using different technologies. In some embodiments, each receiver 208,228 is a Mobile Industry Processor Interface (MIPI) receiver, andtransmitter 218 is a MIPI transmitter. That is, receivers 208, 228 andtransmitter 218 operate according to a MIPI specification. But in otherembodiments, receivers 208, 228 and transmitter 218 may operateaccording to other specifications.

In some embodiments, while the image sensor arrays 202, 212 may have thesame optical format, one or more of the image sensor arrays 202, 212 maypossess different image capture characteristics. These image capturecharacteristics may include, for example, pixel size, sensitivity, readnoise, gain, exposure time, spectrum, other image capturecharacteristics, any combination thereof, and the like. FIG. 3illustrates two image sensor arrays having the same optical format butdifferent image capture characteristics. Referring to FIG. 3, two imagesensor arrays 302, 304 are illustrated. Both image sensor arrays 302,304 have the same optical format. That is, both image sensor arrays 302,304 have the same size and shape. However, the image sensor arrays 302,304 have different pixel sizes. In particular, the pixels of the imagesensor array 304 are four times as large as the pixels of the imagesensor array 302.

By having multiple image sensor arrays 202, 212 with different imagecapture characteristics, an image of a scene may be selected from amongmultiple images having different qualities. In the example of FIG. 3,the image sensor array 304 having larger pixel size will possess highersensitivity, and less read noise, than the image sensor array 302 havingsmaller pixel size. Accordingly, for dark scenes, the image produced bythe image sensor array 304 having larger pixel sizes will have a higherquality than the image produced by the image sensor array 302 havingsmaller pixel size. The image sensor array 302 having smaller pixel sizewill have higher resolution and a large charge handling capacity,meaning the pixels are not as easily saturated. The image sensor array302 having smaller pixel size therefore will produce a higher qualityimage for a well lit scene than the image produced by the image sensorarray 304 having larger pixel sizes. To further optimize these imagesensor arrays 302, 304 for light and dark scenes, the image sensor array302 having smaller pixel size may be implemented with a longer exposure,while the image sensor array 304 having larger pixel sizes may beimplemented with a shorter exposure.

Referring again to FIG. 2, responsive to receiving light 210, theinternal image sensor array 202 may generate image data, and may becoupled to the image selection circuit 222 to provide that generatedimage data to the image selection circuit 222. Similarly, each externalimage sensor array 212 may generate image data responsive to receivingthe light 210. Each external image sensor array 212 may provide itsimage data to a corresponding receiver 208 within the image sensor 214.The receivers 208 a through 208 n may receive the image data into theimage sensor 214, and may be coupled to the image selection circuit 222to provide that received image data to the image selection circuit 222.In accordance with the image selection criteria 220, the image selectioncircuit 222 may select one of the images captured by the image sensorarrays 202, 212. The image selection circuit 222 may be coupled to thetransmitter 218 to provide the image data for the selected image to thetransmitter 218, which may transmit the image data from the image sensor214.

The receiver 228 of the image processor 204 may receive the data for theselected image into the image processor 204, and may provide the imagedata to the image signal processor 206. The image signal processor 206may process the image data according to any image processing algorithm.The image processor 204 may provide the processed image data to thedisplay 216, which may display the processed selected image inaccordance with the processed image data.

Embodiments of the disclosed technology feature several advantages.Compared with the conventional implementation of FIG. 1, the disclosedtechnology, for example as shown in FIG. 2, requires only one receiverto be implemented within the image processor 204. Fewer receivers meansfewer data lines, fewer clock lines, and the like, thereby decreasingthe complexity and cost of the image processor 204. Furthermore, theimage selection function implemented by the image selection circuit 222is now performed outside the image processor 204, further reducing thecomplexity and cost of the image processor 204. Using image sensorarrays 202, 212 configured to capture lights of differing spectrafeatures advantages as well. For example, an infrared image sensor arraymay produce an image showing structural features that differ from thosein an image produced by a visible light image sensor array. The imageselection circuit 222 may select the image having the most usefulfeatures, for example for facial recognition in a high securityenvironment.

FIG. 4 illustrates a process that may be performed by the imaging system200 of FIG. 2 according to embodiments of the disclosed technology.Although elements of the process are depicted in a certain sequence, inother embodiments the elements may be performed in other sequences,concurrently, or any combination thereof. In some embodiments, one ormore elements may be omitted. Referring to FIG. 4, the process 400 mayinclude generating, in the internal image sensor array 202 of the imagesensor 214, first image data for a first image, responsive to receivingthe light 210, at 402.

The process 400 may include receiving, into the image sensor 214, secondimage data for a second image, at 404. The second image data may begenerated by one or more of the external image sensor arrays 212. Theinternal and external image sensor arrays 202, 212 may capture imagessubstantially concurrently. For example, each captured image maycorrespond to a frame in a sequence of frames, and the image sensorarrays 202, 212 may have the same frame rate and phase. The images maybe of the same subject, or of different subjects. The images may includethe same scene, or different scenes.

The process 400 may include selecting one of the first image data andthe second image data according to one or more image selection criteria220, and at least one of the first image data in the second image data,at 406. The image selection criteria 220 may include, for example,noise, dynamic range, resolution, brightness, a quantity of features, aquality of features, a presence of text, facial recognition, facialexpression recognition, other image selection criteria, any combinationthereof, and the like. Process 400 may include transmitting the selectedone of the first image data and the second image data from the imagesensor 214, at 408.

In some embodiments, the imaging system 200 of FIG. 2 may be implementedwithin an automobile, with the display 216 visible to the driver and/orpassengers, and with multiple image sensor arrays 202, 212 capturingimages of scenes ahead of the vehicle. In bright daylight, the imageselection circuit 222 may select an image produced by an image sensorarray 202, 212 having small pixel size, low resolution, low gain, andlonger exposure time. But as the vehicle enters a dark tunnel, the imageselection circuit 222 may select an image produced by an image sensorarray 202, 212 having large pixel size, high resolution, high gain, andshorter exposure time.

Continuing the example with the imaging system 200 of FIG. 2 implementedwithin an automobile, while some of the image sensor arrays 202, 212 maybe arranged to capture images of scenes ahead of the vehicle, anotherimage sensor array 202 may be arranged to capture images of an infant inthe back seat of vehicle. While the images of the scenes ahead of thevehicle may be selected by default, the scene of the infant may beselected according to facial expression recognition criteria when theface of the infant exhibits distress.

It will also be appreciated by persons of ordinary skill in the art thatthe present disclosure is not limited to what has been particularlyshown and described hereinabove. Rather, the scope of the presentdisclosure includes both combinations and sub-combinations of thevarious features described hereinabove as well as modifications andvariations which would occur to such skilled persons upon reading theforegoing description. Thus the disclosure is limited only by theappended claims.

1. An imaging system comprising: an image sensor comprising: a firstimage sensor array to sense light in a first spectrum, and to generatefirst image data based on the sensed light in the first spectrum, areceiver to receive, into the image sensor, second image data, an imageselection circuit to receive the first image data from first imagesensor array, to receive the second image data from the receiver, and toselect one of the first image data or the second image data according toone or more image selection criteria, and a transmitter to transmit theselected one of the first image data or the second image data from theimage sensor; and a second image sensor array to sense light in a secondspectrum, and to generate the second image data based on the sensedlight in the second spectrum, wherein the first spectrum and the secondspectrum are different, and wherein the second image sensor array isexternal to the image sensor.
 2. The imaging system of claim 1, furthercomprising: an image signal processor to process the selected one of thefirst image data or the second image data into a processed image.
 3. Theimaging system of claim 2, further comprising: a display to display theprocessed image.
 4. The imaging system of claim 1, wherein: the firstimage data and the second image data are generated substantiallyconcurrently.
 5. The imaging system of claim 1, wherein the first andsecond image sensor arrays possess different image capturecharacteristics, wherein the image capture characteristics include atleast one of: pixel size; sensitivity; read noise; and exposure time. 6.The imaging system of claim 1, wherein the one or more image selectioncriteria include at least one of: noise; dynamic range; resolution;brightness; a quantity of features; a quality of features; a presence oftext; facial recognition; and facial expression recognition.
 7. Theimaging system of claim 1, wherein: the receiver is a Mobile IndustryProcessor Interface (MIPI) receiver; and the transmitter is a MIPItransmitter.
 8. An image sensor comprising: a first image sensor arrayto sense light in a first spectrum, and to generate first image databased on the sensed light in the first spectrum; a receiver to receive,into the image sensor, second image data generated by a second imagesensor array by sensing light in a second spectrum, wherein the firstspectrum and the second spectrum are different, and wherein the secondimage sensor array is external to the image sensor; an image selectioncircuit to receive the first image data from the first image sensorarray, to receive the second image data from the receiver, and to selectone of the first image data or the second image data according to one ormore image selection criteria; and a transmitter to transmit theselected one of the first image data or the second image data from theimage sensor.
 9. The image sensor of claim 8, wherein: the first imagedata and the second image data are generated substantially concurrently.10. The image sensor of claim 9, wherein: the first image datarepresents a scene; and the second image data represents the scene. 11.The image sensor of claim 8, wherein the first image sensor array andthe second image sensor array possess different image capturecharacteristics, wherein the image capture characteristics include atleast one of: pixel size; sensitivity; read noise; and exposure time.12. The image sensor of claim 8, wherein the one or more image selectioncriteria include at least one of: noise; dynamic range; resolution;brightness; a quantity of features; a quality of features; a presence oftext; facial recognition; and facial expression recognition.
 13. Theimage sensor of claim 8, wherein: the receiver is a Mobile IndustryProcessor Interface (MIPI) receiver; and the transmitter is a MIPItransmitter.
 14. A single integrated circuit comprising the image sensorof claim
 8. 15. A method for an image sensor, the method comprising:generating, in a first image sensor array of the image sensor, firstimage data based on light sensed by the first image sensor array in afirst spectrum; receiving, into the image sensor, second image datagenerated by a second image sensor array based on light sensed by thesecond image sensor array in a second spectrum, wherein the firstspectrum and the second spectrum are different, and wherein the secondimage sensor array is external to the image sensor; selecting one of thefirst image data or the second image data according to one or more imageselection criteria; and transmitting the selected one of the first imagedata or the second image data from the image sensor.
 16. The method ofclaim 15, wherein: the first image data and the second image data aregenerated substantially concurrently.
 17. The method of claim 16,wherein: the first image data represents a scene; and the second imagedata represents the scene.
 18. The method of claim 15, wherein the firstimage sensor array and the second image sensor array possess differentcharacteristics, wherein the characteristics include at least one of:pixel size; sensitivity; read noise; and exposure time.
 19. The methodof claim 15, wherein the one or more image selection criteria include atleast one of: noise; dynamic range; resolution; brightness; a quantityof features; a quality of features; a presence of text; facialrecognition; and facial expression recognition.
 20. The method of claim15, wherein: the second image data are received according to a MobileIndustry Processor Interface (MIPI) specification; and the selectedimage data are transmitted according to the MIPI specification.